Method for storing and retrieving digital image data from an imaging array

ABSTRACT

A method for storing digital information from an image sensor comprises the steps of providing an image sensor producing three-color output data at each of a plurality of pixel locations; providing a digital storage device coupled to the image sensor; sensing three-color digital output data from the image sensor; and storing said three-color output data as digital data in the digital storage device without performing any interpolation on the three-color output data. The data may be compressed prior to storage and expanded after retrieval from storage. In a preferred embodiment, the image sensor comprises a triple-junction active pixel sensor array.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to the capture, storage, and retrieval ofdigital images. More particularly, the present invention relates tonovel methods for storing and retrieving pixel data from a full-colorRGB imaging array imbedded in a device such as a digital camera.

Furthermore, the present invention also relates to active pixel sensorsand active pixel sensor arrays. More particularly, the present inventionrelates to arrays of active pixel sensors wherein each of the activepixel sensors is a triple-junction structure to ensure that each pixelsensor in the array measures each of the three primary colors (R-G-B) inthe same location.

Finally, the present invention relates to a device such as a digitalcamera that employs both a triple-junction active pixel sensor array anda novel method of capturing, storing, and retrieving the data providedby the array.

2. The Prior Art

The process of capturing, processing, storing, and retrieving digitaldata is common in the field of digital imaging.

Generally, a digital image is provided from a source, such as a camera.Many types of cameras are prevalent in the field of digital imaging,including digital, video, and television cameras. Whatever the type ofcamera used, it is often desired that the image be captured and storedin a digital format, so that the image may later be edited or otherwiseprocessed. In the prior art, it is common to interpolate and compressthe digital image data prior to storage. Manipulating the data beforestoring it posses certain disadvantages that are inherent in theprocedures utilized heretofore in the prior art.

First, the process of interpolation may introduce irreversible changesin the digital image data. Interpolation is the process of correctingthe data for errors that occur by virtue of the type of camera or sensorthat is utilized within the camera. Therefore, the type of interpolationthat is used, or the need for interpolation at all, is determined by thenature of the imaging process utilized. For example, it is common in theart to utilize digital sensors that contain charge-coupled devices (CCD)or metal oxide semiconductor (MOS) transistors. Within the sensor, thesmallest resolvable full-color image component (“pixel”) is usuallycomprised of four separate sensors: two green, one blue, and one red.These sensors are used to produce three-color digital output. However,Interpolation is necessary to correct for distortions caused by thesmall, though finite distances, that separate the four individualsensors that make up each pixel. The result of this interpolation isoften a great increase in the size of the original digital image; oftenthis increase in data size is three-fold. Along with this increase insize, interpolation can compromise the integrity of the original data ifperformed prior to storage.

Second, after the step of interpolation, the digital image data is oftencompressed prior to storage. Compression is necessary often because ofthe increase in size after the interpolation function just discussed, aswell as to facilitate transmission through systems of limited bandwidth,such as television systems. However, in compression methods commonlyused, once a digital image has been compressed, it can never be restoredto its original state. This is a major disadvantage if access to theoriginal, uncompressed digital image data is ever desired.

The problems with the interpolation and compression of digital imagedata prior to storage manifest themselves as poor-quality output whenviewed on a screen or printed. In fact, interpolation or compressiontechniques often create moiré patterns on fine-pitched fabrics, orresult in the loss of detail and/or distortions along the edges orbetween fine lines in the subject matter.

In light of the above background, it is apparent that there is a needfor a digital imaging storage and retrieval method that eliminates theproblems associated with the interpolation and compression of digitalimage data.

Furthermore, in light of the above background, it would be advantageousfor a digital imaging storage and retrieval system to be coupled with anactive pixel sensor array.

BRIEF DESCRIPTION OF THE INVENTION

A method according to the present invention for storing digitalinformation from an image sensor array comprising the steps of:providing an image sensor array producing three-color output data ateach of a plurality of pixel locations; providing a digital storagedevice coupled to the image sensor array; sensing three-color outputdata from the image sensor array; and storing the three-color digitaloutput data as digital data in the digital storage device withoutperforming any interpolation on the three-color output data. The storagestep may be performed using a semiconductor memory device such as arandom access memory or the like.

Another method according to the present invention utilizes the abovemethod on an image obtained from an image sensor array furthercomprising a triple-junction structure where each pixel in the arraymeasures each of the primary colors at the same location.

The method of the present invention may also optionally include the stepof performing a lossless compression operation on the three-colordigital output data prior to the step of storing the three-color digitaloutput data in the digital storage device.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

FIG. 1 illustrates the well-known Bayer color filter array (CFA)pattern.

FIG. 2 illustrates the Nyquist domains for red, green and blue resultingfrom the Bayer CFA of FIG. 1.

FIG. 3 is a partial cross-section drawing illustrating a conventionaltwin-well CMOS structure.

FIG. 4 is a partial cross-section drawing illustrating a conventionaltriple-junction CMOS structure.

FIG. 5 is a block diagram of an imager suitable for use with theembodiments of active pixel sensors according to the present invention

FIG. 6 is a schematic diagram of an N-channel MOS implementation of aknown active pixel sensor circuit having a single storage node.

FIG. 7 is a timing diagram illustrating the operation of the activepixel sensor depicted in FIG. 6.

FIG. 8 is a graph plotting light absorption length in silicon versuswavelength.

FIG. 9 is a partial cross-section drawing illustrating a three-colorpixel sensor using a triple-junction structure in accordance with theconcepts of the present invention.

FIG. 10 is a graph showing a set of estimated sensitivity curves for theFIG. 8 triple-junction photodiode structure in accordance with thepresent invention.

FIGS. 11, 12, 13, 14, and 15 are schematic diagrams of active pixelsensors having multiple storage nodes according to first through fifthembodiments of the present invention.

FIGS. 16A and 16B are alternative timing diagrams for the operation ofthe active pixel sensors depicted in FIG. 15 according to the presentinvention.

FIG. 17 is a block diagram of a prior-art image capture and displaysystem showing the interpolation step, the lossy compression step, andthe data storage, data retrieval, and decompression steps performed onthree-color digital output data from the imaging array.

FIGS. 18A and 18B are block diagrams of alternate embodiments of animage capture and display system and method without compressionaccording to the present invention.

FIGS. 19A and 19B are block diagrams of alternate embodiments of animage capture and display system and method using compression accordingto the present invention.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

Persons of ordinary skill in the art will realize that the followingdescription of the present invention is illustrative only and not in anyway limiting. Other embodiments of the invention will readily suggestthemselves to such skilled persons having the benefit of thisdisclosure.

A full RGB imager suitable for use in the present invention is describedin co-pending application Ser. No. 09/290,361, filed on Apr. 12, 1999.Therein, an active pixel imaging array comprising a triple-junctionstructure is described. The advantage of a triple-junction structure isthat each pixel in the array measures each primary color at the samelocation, thus minimizing or eliminating the need for interpolation.

A further advantage of using a full RGB imager is that all of the red,green and blue image information is captured for a single pixel iscontained within a smaller space than in the pixel cluster of prior-artimaging systems, allowing for finer resolution. In a typical systemaccording to the present invention, the full RGB imager may consist of,for example, an array of 640 by 480 three-layer RGB full-color pixelsensor, delivering a total of M=921,600 individual bytes of pixel datain the image dataset. An illustrative non-limiting example of a moredense imager that may be used according to this aspect of the presentinvention is an imager that may consist of an array of 3,000 pixelsensors by 2,000 pixel sensors, (×3 for R,G,B) for a total ofM=18,000,000 bytes of pixel data in the image dataset. An alternativeimplementation of a full RGB imager is an assembly of three simplesensor arrays in a three-color color separation prism with the threearrays optically aligned to each other as is known in the art of videocameras.

The full RGB imager utilized in the present invention is directed tocolor separation in an active pixel MOS imaging array utilizing atriple-junction pixel cell structure to take advantage of thedifferences in absorption length in silicon of light of differentwavelengths to measure different colors in the same location withsensitive areas almost as large as their spacing.

In the present invention a color photosensor structure that separatesblue, green and red light is formed in a P-type silicon body. The colorphotosensor structure comprises a vertical PNPN device that implements atriple stacked photodiode and includes a first N-doped region formed inthe P-type silicon body, a P-doped region formed in the first N-dopedregion, and a second N-doped region formed in the P-doped region. Atriple well process is employed according to the present invention tofabricate the color photosenor structure. The normal N well of thetriple well CMOS process is not employed in the color photosenorstructure of the present invention, although it may be useful to use iton the same chip, outside of the array of imager cells.

In the color photosensor structure, the pn junction formed between theP-type silicon body and the first N-doped region defines a red-sensitivephotodiode at a depth in the silicon approximately equal to theabsorption length of red light in silicon, the pn junction formedbetween the first N-doped region and the P-doped region defines agreen-sensitive photodiode at a depth in the silicon approximately equalto the absorption length of green light in silicon, and the pn junctionformed between the P-doped region and the second N-doped region definesa blue-sensitive photodiode at a depth in the silicon approximatelyequal to the absorption length of blue light. Sensing circuitry isconnected to the red, green and blue photodiodes to integrate and storerespective photodiode currents.

The full RGB imager utilized in the present invention reduces coloraliasing artifacts by ensuring that all pixels in an imaging arraymeasure red, green and blue color response in the same place in thepixel structure. Color filtration takes place by making use of thedifferences in absorption length in silicon of the red, green and bluelight.

The full RGB imager utilized in the present invention providesadvantages in addition to reduction of color aliasing. For example, iteliminates the complex polymer color filter array process steps commonin the prior art. Instead, a triple-junction process, which is commonlyavailable in the semiconductor industry is used. Also, overallefficiency of use for available photons is increased. With thetraditional approach, photons not being passed by the filter materialare absorbed in the filter and wasted. With the approach of the presentinvention, the photons are separated by absorption depth, but are allcollected and used. This can result in an overall improvement in quantumefficiently by around a factor of three.

The full RGB imager utilized in the present invention provides anexcellent example of an imager that would be difficult to implement withconventional CCD technology. In addition, the present invention benefitsfrom the availability of scaled CMOS processing, in the sense that thereare many support transistors in each three-color pixel.

Semiconductor devices for measuring the color of light are known in thenon-imaging art. These devices have been built with a variety oftechnologies that depend upon the variation of photon absorption depthwith wavelength. Examples are disclosed in U.S. Pat. No. 4,011,016,enitled “Semiconductor Radiation Wavelength Detector” and U.S. Pat. No.4,309,604, entitled “Apparatus for Sensing the Wavelength and Intensityof Light.” Neither patent discloses either a structure for a three-colorintegrated circuit color sensor or an imaging array.

In the imaging art, CCD devices with multiple buried channels foraccumulating and shifting photocharges are known. These devices aredifficult and expensive to manufacture and have not been practical forthree-color applications. U.S. Pat. No. 4,613,895, entitled “ColorResponsive Imaging Device Employing Wavelength Dependent SemiconductorOptical Absorption” discloses an example of such a device. This categoryalso includes devices that use layers of thin-film photosensitivematerials applied on top of an imager integrated circuit. Examples ofthis technology are disclosed in U.S. Pat. No. 4,677,289, titled “ColorSensor” and U.S. Pat. No. 4,651,001, entitled “Visible/Infrared ImagingDevice with Stacked Cell Structure.” These structures are also difficultand expensive to make, and have not become practical.

Also known in the imaging art are color imaging integrated circuits thatuse a color filter mosaic to select different wavelength bands atdifferent photosensor locations. U.S. Pat. No. 3,971,065, entitled“Color Imaging Array”, discloses an example of this technology. Asdiscussed by Parulski et al., “Enabling Technologies for Family ofDigital Cameras”, 156/SPIE Vol. 2654, 1996, one pixel mosaic patterncommonly utilized in digital cameras is the Bayer color filter array(CFA) pattern.

Shown in FIG. 1, the Bayer CFA has 50% green pixels arranged in acheckerboard. Alternating lines of red and blue pixels are used to fillin the remainder of the pattern. As shown in FIG. 2, the Bayer CFApattern results in a diamond-shaped Nyquist domain for green andsmaller, rectangular-shaped Nyquist domains for red and blue. The humaneye is more sensitive to high spatial frequencies in luminance than inchrominance, and luminance is composed primarily of green light.Therefore, since the Bayer CFA provides the same Nyquist frequency forthe horizontal and vertical spatial frequencies as a monochrome imager,it improves the perceived sharpness of the digital image.

These mosaic approaches are known in the art to be associated withsevere color aliasing problems due to the facts that the sensors aresmall compared to their spacing, so that they locally sample the imagesignal, and that the sensors for the different colors are in differentlocations, so that the samples do not align between colors. Imagefrequency components outside of the Nyquist domain are aliased into thesampled image with little attenuation and with little correlationbetween the colors.

As pointed out above in the discussion of CCD color imaging arrays, thesemiconductor processes employed in manufacturing the arrays can be bothdifficult and expensive to implement. There are, however, CMOStechnologies that are known which may be implemented with less expenseand greater ease.

Referring to FIG. 3, many modern CMOS integrated circuit fabricationprocesses use a “twin-well” or “twin-tub” structure in which a P wellregion 10 and a N well region 12 of doping density approximately 10¹⁷atoms/cm³ are used as regions within which to make N-channel andP-channel transistors, respectively. The substrate material 14 istypically a more lightly doped P-type silicon (10¹⁵ atoms/cm³), so the Pwell 10 is not isolated from the substrate 14. The N-channel FET 16formed in the P well 10 includes N+normal source/drain diffusions 18 ata dopant concentration of >10⁸ atoms/cm³ and N-type shallow lightlydoped diffusion (LDD) regions 20 at a concentration of approximately10¹⁸ atoms/cm³. The P-channel FET 22 formed in the N well region 12 issimilarly constructed using normal P+source/drain regions 24 and shallowLDD regions 26 of similar dopant concentrations.

Referring to FIG. 4, in an improved process, known as “triple well”, anadditional deep N isolation well 28 is used to provide junctionisolation of the P well 10 from the P substrate 14. The dopant densityof the N isolation well 28 (10¹⁶ atoms/cm³) lies between the dopantdensities of P substrate 14 and P well 10 (10¹⁵ atoms/cm³ and 10¹⁷atoms/cm³, respectively). U.S. Pat. No. 5,397,734, entitled “Method ofFabricating a Semiconductor Device Having a Triple Well Structure”,discloses an example of triple well technology.

Triple well processes are becoming popular and economical formanufacturing MOS memory (DRAM) devices, since they provide effectiveisolation of dynamic charge storage nodes from stray minority carriersthat may be diffusing through the substrate.

Storage pixel sensors are also known in the art. In a storage pixel,data representing intensity of light received by a phototransducer arestored in a storage element that can be read out and cleared usingappropriate control circuitry.

FIG. 5 is a block diagram of an active pixel imager 30 suitable for useaccording to the present invention. In the imager 30, the active pixelsensors are arranged in rows and columns in a pixel sensor array 32. Toextract the analog pixel information from the pixel sensor array 32 forprocessing by an analog-to-digital converter (ADC) 34, a row decodercircuit 36, a column sampling circuit 38, and a counter 40 are employed.The row decoder 34 selects rows from the pixel sensor array 32 inresponse to a row enable signal 42 and signals from the counter 40. Thecolumn sampling circuit 38 is also driven from the counter 40 andfurther includes a multiplexer that couples the sampled columns asdesired to the ADC in response to signals from counter 40.

In a typical implementation, the higher-order bits from counter 40 areused to drive the row decoder circuit 36 and the lower-order bits areused to drive column sampling circuit 38 to permit extraction of allpixel information from a row in the pixel sensor array 32 prior toselection of the next row by row decoder circuit 36. Row decoders,column sampling circuits with embedded multiplexers, and counterssuitable for use in the imager 30 are well known to those of ordinaryskill in the art, and will not be described herein in detail to avoidovercomplicating the disclosure and thereby obscuring the presentinvention.

Referring now to FIG. 6, a schematic diagram of a known active pixelsensor 50 with a single embedded storage element is shown. The activepixel sensor 50 is implemented with N-channel MOS transistors. Those ofordinary skill in the art will appreciate that the active pixel sensor50 may otherwise be implemented with all P-channel MOS transistors or acombination of P-channel and N-channel MOS transistors. In active pixelsensor 50, a photodiode 52 has an anode connected to ground and acathode connected to the source of N-Channel MOS reset transistor 54.The drain of N-Channel MOS reset transistor 54 is connected to Vref andthe gate of N-Channel MOS reset transistor 54 is connected to the globalRESET line indicated by reference numeral 44 in FIG. 5. The RESET lineis preferably driven to a voltage at least a threshold above Vref to setthe cathode of the photodiode 52 to Vref.

The cathode of photodiode 52 is also connected to a first source/drainof N-channel MOS transfer transistor 56. A second source/drain ofN-Channel MOS transfer transistor 56 is connected to a first terminal ofa storage element 58 and also to the gate of N-channel MOS readouttransistor 60. A second terminal of the storage element 58 is connectedto reference potential shown as ground. The gate of N-Channel MOStransfer transistor 56 is connected to the global XFR line indicated byreference numeral 46 in FIG. 5. The connection of the secondsource/drain of N-Channel MOS transfer transistor 56 to the firstterminal of storage element 58 and also to the gate of N-Channel MOStransistor 60 forms a storage node 62. The drain of N-channel MOSreadout transistor 60 is connected to Vcc, and the source of N-channelMOS readout transistor 60 is connected to the drain of N-channel MOS rowselect transistor 64. The gate of N-channel MOS row select transistor 64is connected to a ROW SELECT line, one of which is depicted by referencenumeral 48 in FIG. 5, and the source of N-channel MOS row selecttransistor 64 is connected to a column output line.

It should be appreciated that associated with the storage node 62 arethe N-channel MOS transfer transistor 56 to isolate the storage node 62from further collection of photocharge by the cathode of photodiode 52when an integration period to be described below has ended, theN-channel MOS readout transistor 60 to sense the charge accumulated atstorage node 62, and the storage element 58 to store charge. Further, asdisclosed in co-pending application Ser. No. 09/099,116, entitled“ACTIVE PIXEL SENSOR WITH BOOTSTRAP AMPLIFICATION”, by inventors R. B.Merrill and Richard F. Lyon, filed on Jun. 17, 1998, and assigned to thesame assignee as the present invention, and expressly incorporatedherein by reference, the storage element 58 may be omitted and chargestored on the gate of N-channel MOS readout transistor 60 or that othercapacitive means of charge storage may be employed.

To better understand the operation of the active pixel sensor 50, thetiming diagram of FIG. 7 illustrates the timing of the RESET, XFR andROW SELECT signals depicted in FIG. 6. The active pixel 50 is reset byturning on both N-channel MOS reset transistor 54 and N-channel MOStransfer transistor 56 as shown by the HIGH level of both the RESET andXFR signals at 66 and 68. Then the N-channel MOS reset transistor 54 isturned off at the falling edge 70 of RESET 66 so that integration ofphotocurrent from photodiode 52 can begin. The photocurrent integrationperiod is indicated by reference numeral 72.

While N-channel MOS transfer transistor 56 is turned on, the capacitanceof the storage element 58 adds to the capacitance of the photodiode 52during integration, thereby increasing the charge capacity and the rangeof the active pixel sensor 50. This also reduces variation in the pixeloutput due to capacitance fluctuations since gate oxide capacitance fromwhich storage element 58 is formed is better controlled than junctioncapacitance of the photodiode 52.

When the integration is complete (determined by external exposurecontrol), the N-channel MOS transfer transistor 56 turns off at fallingedge 74 of XFR to isolate the voltage level corresponding to theintegrated photocharge onto the storage element 58. Shortly thereafter,the photodiode 52 itself is preferably reset to the reference voltage byagain turning on N-channel MOS reset transistor 54 as indicated byrising edge 76 of RESET. This action will prevent the photodiode 52 fromcontinuing to integrate during the read out process and possiblyoverflowing excess charge into the body, possibly affecting theintegrity of the signal on the storage element 58.

After the N-channel MOS transfer transistor 56 is turned off, the readout process can begin. Each of the active pixel sensors in a row is readwhen a ROW SELECT signal pulse as shown in FIG. 7 is applied to the gateof the N-channel MOS row select transistor 64 in an active pixel sensor60. In the operation of active pixel sensor 50, a voltage related to thevoltage found on storage node 62 is sensed by N-Channel MOS readouttransistor 50 and placed on the column output line when N-channel rowselect transistor 64 is turned on. The XFR signal stays low until all ofthe rows have been read out or another cycle is initiated.

FIG. 8 illustrates the light absorption length in silicon for light inthe visible spectrum. It is well known that the longer the wavelength oflight incident upon a silicon body, the deeper the light will penetrateinto the silicon body before it is absorbed. As depicted, blue lighthaving wavelengths in the range of about 400-490 nm will be absorbed ina silicon body primarily at a depth of about 0.2-0.5 microns, greenlight having wavelengths in the range of about 490-575 nm will beabsorbed in the silicon body at a depth of about 0.5-1.5 microns, andred light having wavelengths in the range of about 575-700 nm will beabsorbed in the silicon at a depth of about 1.5-3.0 microns.

In FIG. 9, according to the present invention, a triple-junction colorphotosensor structure 78 formed in a silicon body 80 of P-typeconductivity (approx. 1015 atoms/cm2) is illustrated. The colorphotosensor structure 78 includes a first N-type doped well region 82(approx. 1016 atoms/cm3) formed in the P-type silicon body 80, a dopedwell region 84 of P-type conductivity (approx. 1017 atoms/cm3) formed inthe first N-doped region 82, and a second doped region 86 of N-typeconductivity (approx. 1018 atoms/cm3) formed as a very shallow NLDD(N-type lightly doped drain) layer in the P-doped region 84.

Three pn junctions exist in the color photosensor structure 78. A firstpn junction exists between the P-type silicon body 80 and the firstN-doped region 82 at a depth of about 1.5 to about 3.0 microns. Thefirst pn junction is preferably formed at the approximate absorptiondepth for red light of about 2 microns. A second pn junction existsbetween the P-doped region 84 and the first N-doped region 82 at a depthbetween about 0.5 to about 1.5 microns. The second pn junction ispreferably formed at the approximate absorption depth for green light ofabout 0.6 microns. A third pn junction exists between the P-doped region84 and the second N-doped region 86 at a depth of about 0.2 to about 0.5microns. The third pn junction is preferably formed at the approximateabsorption depth for blue light of about 0.2 microns. Accordingly, inthe color photosensor structure 78, the first pn junction forms ared-sensitive photodiode, the second pn junction forms a green-sensitivephotodiode, and the third pn junction forms a blue-sensitive photodiode.

Those skilled in the art will appreciate that the sensitive depletionregions of the diodes described above extends somewhat above and belowtheir junction depths. Such skilled persons will also appreciate thatthe above-described triple-junction structure can be implemented usingregions of opposite conductivities than disclosed in the example of FIG.9, i.e., a first P-doped region in an N-type silicon substrate, anN-doped region in the first P-region and a second P-doped region in theN-region. However, such a structure is usually not used in the industry,and the structure of FIG. 9 is preferred since it uses standardtriple-junction MOS memory technology. Additionally, persons of ordinaryskill in the art will appreciate that additional pn junctions could beformed at selected depths in the color photosensor structure 78 byforming additional doped regions to provide for the absorption ofphotons at additional selected wavelengths.

FIG. 9 further shows that the color photosensor structure of the presentinvention also includes a sensing mechanism 88 connected to the red,green and blue photodiodes for measuring red, green and bluephotocurrents, respectively, across the three photodiodes. Thephotocurrent sensor 88 is illustrated as a conceptual arrangement thatincludes a first current meter 90 connected across the red-sensitivephotodiode for measuring the red photocurrent ir. A second current meter92 is connected across the green-sensitive photodiode for measuring thegreen photocurrent ig. A third current meter 94 is connected across theblue-sensitive photodiode for measuring the blue photocurrent ib.Assuming that most of the current in the photodiodes is collected intheir depletion regions, those skilled in the art will clearlyappreciate that the current ib will be primarily photocurrent ofincident photons from the blue end of the visible spectrum, the currentig will be primarily current from green photons, and the current ir willbe primarily current from red photons.

FIG. 10 presents a set of estimated sensitivity curves for the triplestacked photodiode arrangement of the present invention, as a functionof wavelength within the visible spectrum. The curves are only ratherbroadly tuned, as shown, rather than sharply tuned as in some othercolor separation approaches that are based on color filters. However, asis well known in the art of color imaging, it is possible with suitablematrixing to convert three measurements from such a set of curves into amore nearly colorimetrically correct set of red, green, and blueintensity values. Methods for estimating suitable matrix transformationsare known in the art, and are disclosed, for example in U.S. Pat. No.5,668,596, entitled “Digital Imaging Device Optimized for ColorPerformance.”

According to the present invention, an imager 30 such as thatillustrated in FIG. 5 has multiple storage nodes associated with each ofthe pixels in the pixel array 32. To capture a color image in the imager30, each of the pixels employs the triple-photodiode color sensorstructure 78 described with reference to FIG. 9. In each of theembodiments according to the present invention of the storage pixelsensors 100-1 through 100-5 depicted in FIGS. 11 through 15 herein, eachof the three diodes in the triple-diode color photosensor structure 78has a terminal that is coupled to at least one separate storage andreadout circuit. The embodiments of storage pixel sensors 100-1 through100-5 depicted in FIGS. 11 through 15 are implemented with N-channel MOStransistors. Those of ordinary skill in the art will appreciate that thestorage pixel sensors below may otherwise be implemented with P-channelMOS transistors or a combination of N-channel and P-channel MOStransistors. Corresponding elements depicted in FIGS. 11 through 15 willbe identified by the same reference numerals.

In the operation of the active pixel sensors 100-1 through 100-4 ofFIGS. 11 through 14, the active pixel sensors are reset and charge isaccumulated in a manner similar to that described above with respect tothe pixel sensor of FIG. 6. For the operation of active pixel sensor100-5, alternative timing diagrams are depicted in FIGS. 16A and 16B.

In each of the embodiments of the active pixel sensors 100-1 through100-5, the first N-doped region 82 is coupled to a source of N-channelMOS reset transistor 102-1, the P-doped region 84 is coupled to a drainof N-channel MOS reset transistor 102-2, and the second N-doped region86 is coupled to a source of N-channel MOS reset transistor 102-3. Thedrains of N-channel MOS reset transistors 102-1 and 102-3 are coupled toreference voltage Vn, and the source of N-channel MOS reset transistor102-2 is coupled to a reference voltage Vp<Vn. The gates of N-channelMOS reset transistors 102-1 and 102-3 are connected to a RESET-N controlline 104, and the gate of N-Channel MOS reset transistor 102-2 isconnected to a RESET-P control line 106.

The potential Vn coupled to the drains of N-channel MOS resettransistors 102-1 and 102-3 are substantially positive with respect tothe P-type silicon substrate, and the potential Vp coupled to the drainof N-Channel MOS reset transistor 102-2 is less positive than Vn, sothat all three photodiodes start out in a reverse biased state when theRESET-N and RESET-P signals are applied. As the photodiodes in thetriple-diode color photosensor structure 78 are exposed to light, theybecome less reverse biased, and can even become somewhat forward biasedbefore they “overflow.” The three voltages sensed will correspond todifferent linear combinations of the photocharges, depending on thevalues of the various photodiodes and stray capacitances of the circuit.These linear combinations affect the resulting sensitivity curves forthe voltage output and, hence, are corrected for in the matrixtransformation that follows to produce a calorimetrically sensible finaloutput.

Further, the active pixel sensors 100-1 through 100-5 each include aplurality of storage nodes 108-1, 108-2 and 108-3. For example, storagenode 108-1 comprises the common connection of the first terminal of astorage element 110-1, a first source/drain of N-channel MOS transfertransistor 112-1, and the gate of N-channel MOS readout transistor114-1. Storage node 108-2 comprises the common connection of the firstterminal of storage element 110-2, a first source/drain of N-channel MOStransfer transistor 112-2, and the gate of N-channel MOS readouttransistor 114-2. Storage node 108-3 comprises the common connection ofthe first terminal of a storage element 110-3, a first source/drain ofN-channel MOS transfer transistor 112-3, and the gate of N-channel MOSreadout transistor 114-3. The gates of N-channel MOS transfertransistors 112-1, 112-2 and 112-3 are connected to a global transfersignal on XFR line 116. The storage elements 110-1, 110-2, and 110-3each have a second terminal connected to a fixed potential shown asground.

Referring to the embodiment 100-1 of an active pixel sensor according tothe present invention as seen in FIG. 11, voltages present on storagenodes 108-1 through 108-3 are read out onto separate column output lines118-1 through 118-3, respectively, by a single row select signal on ROWSELECT line 120. Accordingly, the drain of each N-channel MOS readouttransistor 114-1 through 114-3 is connected to Vcc, and the source ofeach N-channel MOS readout transistor 114-1 through 114-3 is connectedto the drain of one of N-channel MOS row select transistors 122-1through 122-3, respectively. The gates of N-channel MOS row selecttransistors 122-1 through 122-3 are each connected to the ROW SELECTline 120, and the sources of N-channel MOS row select transistors 122-1through 122-3 are connected to the column output lines 181-1 through118-3, respectively.

In the operation of the active pixel sensor 100-1, during the readingout of the images on the column output lines 118-1 through 118-3, columncircuits (not shown) connected to the column output lines 118-1 through118-3, respectively, may be used to select a pixels representing astored image provided on the storage nodes 108-1 through 108-3. Further,column circuits may be used to perform some function on the storedpixels, such as performing a linear combination of the sensed colorsignals.

Referring now to FIG. 12, in the embodiment 100-2 of an active pixelsensor according to the present invention, voltages present on thestorage nodes 108-1 through 108-3 are read out separately onto the samecolumn output line 118, by separately asserting ROW SELECT1 through ROWSELECT3 signals. Accordingly, the drain of each N-channel MOS readouttransistor 114-1 through 114-3 is connected to Vcc, and the source ofeach N-channel MOS readout transistor 114-1 through 114-3 is connectedto the drain of N-channel MOS row select transistors 122-1 through122-3, respectively. The gates of N-channel MOS row select transistors122-1 through 122-3 are each connected to respective ones of ROW SELECT1through ROW SELECT3 lines 120-1 through 120-3, respectively, and thesources of N-channel MOS row select transistors 122-1 through 122-3 areconnected to the single column output line 118.

In the operation of active pixel sensor 100-2 of FIG. 12, the imagestored on storage node 108-1 will be read out in response to a HIGH ROWSELECT1 signal, the image stored on storage node 108-2 will be read outin response to a HIGH ROW SELECT2 signal, and the image stored onstorage node 108-3 will be read out in response to a HIGH ROW SELECT3signal. It should be understood that the imager 30 depicted in FIG. 5will further include additional decoding circuits for providing thesignals on ROW SELECT1 through ROW SELECT3 lines.

Referring now to FIG. 13, in the active pixel sensor embodiment 100-3,voltages present on storage nodes 108-1 through 108-3 are read outseparately onto a single column output line 118 in response to signalson IMAGE SELECT1 through IMAGE SELECT3 lines 126-1 through 126-3,applied to the gates of N-channel MOS image select transistors 124-1through 124-3, respectively, and a signal on ROW SELECT line 120.Accordingly, the drains of N-channel MOS readout transistors 114-1through 114-3 are each connected to Vcc, and the sources of N-channelMOS readout transistors 114-1 through 114-3 are connected to the drainsof N-channel MOS image select transistors 124-1 through 124-3,respectively. The gates of N-channel MOS image select transistors 124-1through 124-3 are connected to IMAGE SELECT1 through IMAGE SELECT3 lines126-1 through 126-3, respectively. The sources of N-channel MOS imageselect transistors 124-1 through 124-3 are all connected to the drain ofN-channel MOS row select transistors 128. The gate of N-channel MOS rowselect transistor 128 is connected to a ROW SELECT line 120, and thesource of N-channel MOS row select transistor 128 is connected to acolumn output line 118.

In the operation of active pixel sensor 100-3, the image stored onstorage node 108-1 will be read out in response to a high signalasserted on ROW SELECT line 120 and a high signal asserted n IMAGESELECT1 line 126-1. The image stored on storage node 108-2 will be readout in response to a high signal asserted on ROW SELECT line 120 and ahigh signal asserted on IMAGE SELECT2 line 126-2. The image stored onstorage node 108-3 will be read out in response to a high signalasserted on ROW SELECT line 120 and a high signal asserted on IMAGESELECT3 line 126-3. It should be understood that the imager 30 depictedin FIG. 5 will further include global IMAGE SELECT1 through IMAGESELECT3 lines. The use of the global IMAGE SELECT1 through IMAGE SELECT3signals in combination with the ROW SELECT signal eliminates the needfor the additional row decoding required in the embodiment of FIG. 12.

Referring now to FIG. 14, in the active pixel sensor embodiment 100-4,voltages present on storage nodes 108-1 through 108-3 are read out in acurrent mode onto a single column output line 118 in response to signalsasserted on IMAGE SELECT1 through IMAGE SELECT3 lines 126-1 through126-3, respectively, and a signal asserted on ROW SELECT line 120.Accordingly, the drains of N-channel MOS readout transistors 114-1through 114-3 are connected together, and to the source of an N-channelMOS row select transistor 128. The sources of N-channel MOS readouttransistors 112-1 through 114-3 are connected to IMAGE SELECT1 throughIMAGE SELECT3 lines 126-1 through 126-3, respectively. The gate ofN-channel MOS row select transistor 128 is connected to a ROW SELECTline 120, and the drain of N-channel MOS row select transistor 128 isconnected to a column output line 118.

In the operation of active pixel sensor 100-4, the column output line118 is connected to the drain of the N-channel MOS row select transistor128. To place current representing the stored image on the column outputline 118, the image stored at storage node 108-1 will be selected by alow signal asserted on IMAGE SELECT1 line 126-1, the image stored atstorage node 108-2 will be selected by a low signal asserted on IMAGESELECT2 line 126-2, and the image stored at storage node 108-3 will beselected by a low signal asserted on IMAGE SELECT3 line 126-3. Thecurrent-mode output on column output line 128 is therefore controlled bythe signals on IMAGE SELECT1 through IMAGE SELECT3 lines 126-1 through126-3. The column output line 128 output must be kept biased to a highenough voltage that the non-selected N-channel MOS readout transistors114-1 through 114-3 do not start conducting backward. Further, it shouldbe appreciated that the voltage drivers for the IMAGE Select1 throughIMAGE SELECT3 lines 126-1 through 126-3 must be capable of sinking allthe column current from the selected row.

Referring now to FIG. 15, an active pixel sensor embodiment 100-5 isseen to resemble the embodiment of FIG. 11 and includes additionalstorage nodes to demonstrate that the multiple storage nodes may bematrixed using ROW SELECT1 through ROW SELECT2 and COLUMN OUPUT1 throughCOLUMN OUTPUT3. In most respects, the embodiment of FIG. 15 functions inthe same manner as the embodiment of FIG. 11.

In the active pixel sensor 100-5 shown in FIG. 15, the voltages presenton storage nodes 108-1 through 108-3 are read out onto column outputlines 118-1 through 118-3, respectively, by the signal on ROW SELECT1line 120-1, and the voltages present on storage nodes 108-4 through108-6 are read out onto column output lines 118-1 through 118-3,respectively, by the signal on ROW SELECT2 line 120-2. Accordingly, thedrain of each N-channel MOS readout transistors 114-1 through 114-6 areconnected to Vcc, and the source of each N-channel MOS readouttransistor 114-1 through 114-6 is connected to the drain of an N-channelMOS row select transistor 122-1 through 122-6, respectively. The gatesof N-channel MOS row select transistors 122-1 through 122-3 are eachconnected to the ROW SELECT1 line 120-1, and the gates of N-channel MOSrow select transistors 122-4 through 122-6 are each connected to the ROWSELECT2 line 120-2. The sources of N-channel MOS row select transistors122-1 and 122-4 are connected to first column output line 118-1, thesources of N-channel MOS row select transistors 122-2 and 122-5 areconnected to second column output line 118-2, and the sources ofN-channel MOS row select transistors 122-3 and 122-6 are connected tothird column output line 118-3.

In the operation of the active pixel sensor 100-5, charge stored on anyof the storage nodes 108-1 through 108-6 in is read out in response tothe assertion of signals on either of the ROW SELECT1 and ROW SELECT2lines applied to the gates of N-channels MOS row select transistors122-1 through 122-3 or 122-4 through 122-6 to which the storage nodes108-1 through 106-3 or 108-4 through 108-6, respectively, are coupledand by sensing the column output lines 118-1 through 118-3 to which thestorage nodes are coupled.

For example, to select a pixel information stored on the storage node108-1, the signal on ROW SELECT1 line 120-1 will be asserted and thefirst column output line 118-1 will be sensed. In embodiments wheremultiple storage nodes are employed, the matrixing of the storage nodes108-1 through 108-6 using ROW SELECT1 and ROW SELECT2 lines 120-1 and120-2 and first, second, and third column output lines 118-1, 118-2, and118-3 reduces the number of additional row and column lines required. Itshould also be understood that instead of the single global XFR linedepicted in FIG. 1 that first and second global transfer lines XFR1 andXFR2 (shown at reference numerals 116-1 and 116-2) will be employed,allowing for motion sensing, multiple exposure times, and the like.

FIGS. 16A and 16B are timing diagrams showing the RESET-N, RESET-P, XFR1and XFR2 signals and illustrating the operation of active pixel sensor100-5. In FIG. 16A, with XFR1 signal asserted high on line 116-1, theRESET-N and RESET-P signals (shown for simplicity as a single RESETsignal) make a transition at falling edge 130 to begin the accumulationof charge on storage nodes 108-1, 108-2, and 108-3. The XFR1 signal thenmakes a transition at falling edge 132, halting the accumulation ofcharge on storage nodes 108-1, 108-2, and 108-3. The RESET signal isthen makes a transition at rising edge 134 to reset the voltage of thephotodiodes in the three-diode color photosensor structure 78. The XFR2signal on line 116-2 then makes a transition at rising edge 136. Whenthe RESET signal makes a transition at falling edge 138, accumulation ofcharge on storage nodes 108-4, 108-5, and 108-6 begins. The XFR2 signalon line 116-2 then makes a transition at falling edge 140, halting theaccumulation of charge on storage nodes 108-4, 108-5, and 108-6.

In FIG. 16B, with XFR1 and XFR2 lines both asserted high, the RESETsignal makes a transition at falling edge 150 to begin the accumulationof charge on storage nodes 108-1, 108-2, 108-3, 108-4, 108-5, and 108-6.The XFR1 signal then makes a transition at falling edge 152, halting theaccumulation of charge on storage nodes 108-1, 108-2, and 108-3. Theaccumulation of charge on storage nodes 108-4, 108-5, and 108-6continues. Then XFR2 signal makes a transition at falling edge 154,halting the accumulation of charge on storage nodes 108-4, 108-5, and108-6.

Having now fully described the advantages of a full RGB imager, thereader is now directed to FIG. 17 and the storage and retrieval methoddescribed therein.

Referring now to FIG. 17, a block diagram of a typical prior-art imagecapture and display system is shown. An image is first captured byfilter-mosaic imager 210 having M pixel sensors. Color image sensors inthe prior art typically sense only one of the three primary colors ateach pixel location, through a mosaic of color filters integrated ontothe image sensor chip, in distinction from the full-color sensors of thepresent invention that sense each of the three colors at each pixellocation. In a typical system, the prior-art filter-mosaic imager mayconsist of, for example, an array of 640 pixel sensors by 480 pixelsensors delivering a dataset having a total of M=307,200 bytes of pixeldata. A more dense imager may consist of an array of 3,000 pixel sensorsby 2,000 pixel sensors, for a total of M=6,000,000 bytes of pixel datain the dataset.

The output dataset from pixel sensors in imager 210 is then processed byinterpolator 212 in order to convert it to a full RGB dataset as isknown in the art. The interpolation process increases the size of thedataset to 3M. Color transformations and corrections are then performedon the dataset by color corrector 214, as is known in the art.

After interpolation and color correction have been performed on theoutput pixel dataset from the imager 210, data compression, such as JPEGcompression, is performed on the dataset in data compressor 216. JPEGcompression is an industry standard and results in an adjustable degreeof compression for which 0.25x is a typical example, resulting in adecrease in the size of the dataset to 0.75M as shown in FIG. 17.

After the dataset has been compressed, it may then be stored in storageelement 218. Storage element 218 has taken numerous forms in the priorart, such as magnetic storage (e.g., floppy disks), or digitalsemiconductor memory storage such as flash or random access memory.

When it is desired to display or print a stored digital image, thestored compressed data representing the color-corrected image is firstretrieved from storage element 218 by storage retrieval element 220. Thenature of storage retrieval element 220 depends on the nature of storageelement 218 with which it functions, as is appreciated by persons ofordinary skill in the art.

After the stored dataset representing the image has been retrieved fromstorage element 218 by storage retrieval element 220, it is thendecompressed by decompression element 222 as is known in the art andthen provided to display or printer 224 as required by the user.

The image data storage and retrieval method performed by the system ofFIG. 17 is easily inferred from the block diagram of FIG. 17. The stepsof the image data storage and retrieval method performed by the priorart image capture and display system will be referred to using the samereference numerals that identified the elements performing these steps.Thus, first, at step 210, an image is captured by the imager. Next, atstep 212, the dataset representing the image is interpolated to producea full RBG dataset. The RGB dataset is then processed at step 214 toperform color transformation and correction on the dataset as desired.At step 16 the dataset is compressed and at step 218 the dataset is thenstored.

When it is desired to display or print a stored digital image, thestored compressed dataset representing the color-corrected RGB image isfirst retrieved from storage at step 220. The retrieved compresseddataset representing the image is then decompressed at step 222 as isknown in the art. Finally, at step 224, the image data is then providedto display or printer 224 as required by the user using conventionaltechniques.

The interpolation step 212 and the compression step 216 performed by theprior-art scheme depicted in FIG. 17 are “lossy” in that they representa compromise with respect to the resolution of the original image datain the dataset obtained from the imager 210. These steps areirreversible in that the original dataset taken from the imager 210 isnot recoverable. More importantly, the original dataset from imager 210is incapable of rendering a complete description of the image falling onthe sensor array. It is well known in the art that to achieve a completedescription of an image, the image must be sampled at least twice ineach dimension for each cycle of the highest spatial frequency presentin the optical image. The highest spatial frequency is typically set bythe modulation transfer function of the lens, which for typicalphotographic lenses is on the order of one cycle per 10 micrometers. Thesize of a typical photosensor on a high density imaging array is about 5micrometers, so the sampling criterion is just satisfied. However, witha filter mosaic, the repeated unit used to sample the image consists of4 sensors, and is typically between 10 and 20 micrometers in eachdimension. This large sampling interval inevitably results in anirreversible loss of information by confusing higher spatial frequencieswith lower spatial frequencies; this problem is known as aliasing. Thealias artifacts created by this procedure are seen in digital images asmoire patterns on fine-pitched fabrics, or as colored highlights alongedges and fine lines. The aliasing artifacts are usually preserved andoften accentuated by lossy compression techniques and by attempts tosharpen the image.

FIGS. 18A and 18B are block diagrams of alternate embodiments of animage capture and display system without compression according to thepresent invention. With reference first to FIG. 18A, one embodiment ofan image capture and display system 230 according to the presentinvention is presented.

Image capture and display system 230 preferably includes a full RGBimager 232, i.e., an imager that senses all of the three primary colorsat each pixel location to produce a full RGB image dataset. As will beapparent to one skilled in the art, the full RGB imager 232 mayoptimally be located in an imaging device such as a digital camera.

A full RGB output dataset from pixel sensors in imager 232 is thenprocessed by color corrector 234 to perform color transformations andcorrections. Color corrector 234 may be configured as in the prior artexample shown in FIG. 17 and its structure and operation are thereforefamiliar to persons of ordinary skill in the art. Examples of colortransformations and corrections that may be performed by color corrector234 are dark signal subtraction, matrixing, bad pixel replacement,linearization and gamma encoding. Color correction is optional and neednot be performed according to the present invention if unnecessary.

After color correction has been performed on the RGB dataset from theimager 232 of the present invention, the color-corrected dataset maythen be directly stored in storage element 236. Storage element 236 maytake numerous forms, such as magnetic storage (e.g., floppy disks), ordigital semiconductor memory storage such as flash or random accessmemory. Persons of ordinary skill in the art will observe that otherstorage techniques, such as optical storage, may also be used in thesystem and method of the present invention is not limited to thosestorage techniques specifically enumerated herein.

When it is desired to display or print a stored digital image accordingto the system and method of the present invention, the datasetrepresenting the stored color-corrected image is first retrieved fromstorage element 236 by storage retrieval element 238. Persons ofordinary skill in the art will appreciate that the nature of storageretrieval element 220 depends on the nature of storage element 218 withwhich it functions. As a non-limiting example, if semiconductor memoryis employed in the present invention, the conventional memory addressingand reading circuitry will perform the function of storage retrievalelement 238.

After the dataset representing the stored color-corrected image has beenretrieved from storage element 236 by storage retrieval element 238, itmay then be interpolated by interpolation element 240. According to thepresent invention, interpolation element 240 may perform the process ofinterpolating from sensor resolution to a higher output resolution, forexample to prevent pixel artifacts on a print on the data in the datasetprior to display or printing. Interpolation element 240 may comprise,for example, a microprocessor running interpolation software as would beappreciated by persons of ordinary skill in the art. Persons of ordinaryskill in the art will recognize that the interpolation step is notnecessary to the practice of the present invention.

Finally, the interpolated dataset from interpolation element 240 is thenprovided to display or printer 242 as required by the user or may bestored or transmitted in this higher resolution form for later use orfurther processing, as when a photographer delivers an image to aclient. Hardware and software techniques for providing image data toprinters or displays are well known to persons of ordinary skill in theart.

The image data storage and retrieval method of the present inventionperformed by the system of FIG. 18A is easily inferred from the blockdiagram therein. The steps of the image data storage and retrievalmethod performed by the image capture and display system of FIG. 18Awill be referred to using the same reference numerals that identifiedthe elements performing these steps. Thus, first, at step 232, an imageis captured by the imager and an image dataset is formed. Next, at step234, the image dataset is then processed to perform color transformationand/or correction if desired. At step 236 the dataset is then stored.

When it is desired to display or print a stored digital image, thestored dataset representing the color-corrected image is retrieved fromstorage at step 238. The retrieved dataset representing the storedcolor-corrected image may then be interpolated at step 240 if desired.Finally, at step 242, the image dataset is then provided to display orprinter as required by the user and known to persons of ordinary skillin the art.

As may be observed from an examination of FIG. 18A, the amount of data Min the dataset remains constant throughout the storage and retrievalprocess until the interpolation step 240, where the size of the datasetis increased by the interpolation process. In the example given in FIG.18A, the optional interpolation step increases the amount of data in theimage dataset from M to 4M. Persons of ordinary skill in the art willrecognize that the example shown in FIG. 18A is a non-limiting example,and other interpolation processes performed in accordance with theprinciples of the present invention will result in increasing the amountof data by factors other than 4.

Referring now to FIG. 18B, a variation on the image capture and displaysystem and method of the present invention of FIG. 18A is presented.Because the elements and process steps of the embodiment of FIG. 18B arepresent in the embodiment of FIG. 18A, the same reference numerals usedin FIG. 18A will be employed to identify the corresponding elements andsteps of the embodiment of FIG. 18B.

In the variation of the image capture and display system and method ofthe present invention depicted in FIG. 18B, the full RGB dataset fromimager 232 is stored in storage element 36 without any colortransformation or correction being performed. As may be seen from anexamination of FIG. 18B, the color transformation and/or correction isperformed on the dataset after retrieval from storage at step 238 andprior to interpolation and display or printing. Otherwise, the imagecapture and display system depicted in FIG. 18B may be identical to thatdepicted in FIG. 18A.

The image capture and display method performed by the embodiment of thepresent invention depicted in FIG. 18B starts with the same step 232 ofthe method of FIG. 18A wherein the image data is captured by the imagerand formed into an image dataset. Next, at step 236, the raw imagedataset is stored.

When it is desired to display or print a stored digital image, thedataset representing the stored image is retrieved from storage at step238. Color correction and/or transformation is then performed on theretrieved data at step 234. The dataset representing the color-correctedimage may then be interpolated at step 240 if desired. Finally, at step242, the dataset is then provided to display or printer as required bythe user and known to persons of ordinary skill in the art.

As may be observed from an examination of FIG. 18B, the amount of data Min the image dataset remains constant throughout the storage andretrieval process until the interpolation step 240, where the amount ofdata in the dataset is increased by the interpolation process. In theexample given in FIG. 2B, the optional interpolation step increases theamount of data in the image dataset by a factor of 4 from M to 4M.Persons of ordinary skill in the art will recognize that the exampleshown in FIG. 2B is a non-limiting example, and other interpolationprocesses performed in accordance with the principles of the presentinvention will result in increasing the amount of data in the imagedataset by factors other than 4.

FIGS. 19A and 19B are block diagrams of alternate embodiments of animage capture and display system and method using compression accordingto the present invention. According to the embodiments of FIGS. 19A and19B, the image dataset may be compressed to decrease the system storagerequirements. Because certain of the elements and process steps of theembodiment of FIGS. 19A and 3B are present in the embodiments of FIGS.19A and 19B, the same reference numerals used in FIGS. 19A and 19B willbe employed to identify the corresponding elements and steps of theembodiments of FIGS. 19A and 19B.

Referring now to FIG. 19A, one embodiment of the second image captureand display system 260 using compression according to the presentinvention is presented.

Image capture and display system 260 includes a full RGB imager 232 asdescribed with reference to the previously-described embodiment. A fullRGB output dataset from the pixel sensors in imager 232 is thenprocessed by color corrector 234 to perform color transformations andcorrections on the image dataset. Color corrector 234 may be configuredas in the prior art example shown in FIG. 17 and the embodiments of thepresent invention illustrated in FIGS. 18A and 18B. Color correctionaccording to this embodiment of the present invention is optional andneed not be performed if deemed unnecessary.

After optional color correction has been performed on the image datasetfrom the imager 232 of the present invention, the color corrected imagedataset may then be subjected to a data-compression step in datacompressor 262. The data compression step performed according to thepresent invention in data compressor 232 is a lossless compression,i.e., one such that the stored data can be later decompressed to producethe identical pre-compression data, or a “nearly lossless compressionstep. As will be appreciated by persons of ordinary skill in the art,various means, such as a compression integrated circuit or amicroprocessor running compression software may be used to perform thisfunction.

Compared to prior art methods, the present invention as disclosed hereinprovides a better combination of image quality and data storagerequirements in a system in which quality is a dominant concern. Priorart methods that sense colors through a filter mosaic, then interpolate,and then compress, can achieve a comparable combination of imageresolution and storage requirements, but then suffer from a potentialfor aliasing at the sensor; aliasing is a well-known artifact of sensingthrough a filter mosaic, and can not be fully corrected by subsequentprocessing.

Furthermore, by not interpolating before storage, the present inventionallows the image processing steps such as color correction (matrixing,bad pixel replacement, and such steps) to be done after retrieval of theimage data, and therefore allow for improved or modified processingsteps to be used at retrieval time. Therefore, the image quality is notirretrievably compromised by the processing and correction algorithms atthe time of image capture and storage. Furthermore, since the full RGBimage sensor delivers all three color measurements at each pixellocation, the data can be stored in a standard RGB scanned image formatfile without data interpolation or other expansion operations; thisproperty of the invention allows the data to be stored and retrieved ina standard way such that subsequent processing can be done with standardcolor image processing tools.

In embodiments of the present invention employing compression, the sameadvantages can be retained while further reducing the size of the storeddataset, for example by about half. As an example of using standardcolor image file formats, the TIFF (tagged image file format) standardallows an LZW (Lempel-Ziv-Welch) lossless compression that is compatiblewith standard TIFF file retrieval tools. Since the decompressed datasetmatches exactly the dataset before compression, no loss of quality isnecessary to gain this storage advantage.

Storage of data, such as image datasets, generally involves some kind ofdata precision compromise, such as the number of bits per color perpixel; that compromise is usually viewed a representation issue, ratherthan a compression issue. For example, image sensors generally measurelight intensity and represent the result using 10 to 14 bits in a linearrepresentation of intensity; before delivering that data as an image,however, they most often convert to a nonlinear or gamma-compressedrepresentation and then round to 8 bits per color per pixel. At thislevel of precision and this nonlinear representation, the resulting lossof quality is usually far below a perceptible level. In the presentinvention, the advantage of storing raw data, or color correctedprocessed data, from an RGB imager, can be retained if the dataset isconverted to a conventional 8-bit-per-color-per-pixel representation andstored without compression or with lossless compression.

Furthermore, the same advantages can be obtained by storing the imagedataset using a “nearly lossless” compression technique, especially incases in which the dataset is not first converted to a representationwith a small number of bits per pixel per color. For example, if theimager, or the color corrector, delivers an image using 14 bits perpixel per color, then a nearly lossless compression algorithm can beused on that dataset directly, as long as the retrieved and decompresseddataset is sufficiently close to the original dataset to keep the errorsbelow a perceptible level.

For the purpose of the present invention, an imagecompression/decompression technique is defined to be “nearly lossless”if the error between the original image and the decompressed image isnot more than three times the error of the usual representational stepof converting to 8-bit gamma-compressed data; the errors are measured ina root-mean-square sense on typical images, in which the usualstatistics of quantization give an rms error of ⅓ of an 8-bit LSB stepfor the usual quantization, so allow an error equivalent to 1 LSB stepfor nearly lossless compression/decompression with 8-bit gamma-encodedoutput.

Note that most lossy image compression techniques, including JPEG with aquality setting of “maximum”, lead to larger errors, and so are not inthe class of “nearly lossless” as defined herein. The defining thresholdhas been taken to be approximately the amount of noise added by a fewtypical image processing steps such as minor curves or level adjustmentin a program such as Adobe Photoshop, since these are not usuallyregarded as significantly lossy operations.

The particular type of either lossless or nearly lossless datacompression used with actual embodiments fabricated according to theprinciples of the present invention is largely a matter of designchoice.

After data compression, the compressed image dataset is stored instorage element 236. Storage element 236 may take the same numerousforms, such as magnetic storage (e.g., floppy disks), or digitalsemiconductor memory storage such as flash or random access memory, asin the previously-described embodiments of the invention.

When it is desired to display or print a stored digital image accordingto the system and method of the present invention, the stored datarepresenting the color-corrected image is first retrieved from storageelement 236 by storage retrieval element 238. As with the previouslydescribed embodiments of the present invention, persons of ordinaryskill in the art will appreciate that the nature of storage retrievalelement 220 depends on the nature of storage element 218 with which itfunctions.

Referring back to FIG. 19A, after stored data has been retrieved fromstorage element 236 by storage retrieval element 238, it is expanded ordecompressed in data-expander element 264. The nature of data-expanderelement 264 will depend on the nature of data compression element 262,since these two elements perform functions which are the inverse of oneanother, or nearly so. Data expander technology is well known in theart.

After the retrieved image dataset has been expanded, it may then beinterpolated by interpolation element 240. Interpolation element 240 maybe the same as in the previously described embodiments herein. As withthe previously described embodiment herein, persons of ordinary skill inthe art will recognize that the interpolation step is not necessary tothe practice of the present invention.

Finally, the interpolated image dataset from interpolation element 240is then provided to display or printer 242 as required by the useremploying known hardware and software techniques for providing imagedata to printers or displays.

The image data storage and retrieval method of the present inventionperformed by the system of FIG. 19A is easily inferred from the blockdiagram therein. The steps of the image data storage and retrievalmethod performed by the image capture and display system of FIG. 19Awill be referred to using the same reference numerals that identifiedthe elements performing these steps. Thus, first, at step 232, an imageis captured by the imager and an image dataset is formed. Next, at step234, the image dataset is then processed to perform color transformationand/or correction if desired to produce a color-corrected image dataset.Next, at step 262, lossless or “nearly lossless” data compression isperformed on the color-corrected image datset prior to storage. At step236 the compressed image dataset is then stored.

When it is desired to display or print a stored digital image, thestored data representing the color-corrected image is retrieved fromstorage at step 238. Next, the retrieved data is expanded at step 264.The retrieved data representing the color-correcteded image may then beinterpolated at step 240 if desired. Finally, at step 242, the imagedataset is then provided to display or printer as required by the userand known to persons of ordinary skill in the art.

As may be observed from an examination of FIG. 19A, the amount of datastored is less than that stored in the embodiments of FIGS. 18A and 18B.In the embodiments described in FIGS. 18A and 18B, the amount of data Min the image dataset remains constant throughout the storage andretrieval process until the interpolation step 240, where the amount ofdata may be increased by the interpolation process. In the example givenin FIG. 3A, the data compression step decreases the amount of data inthe image dataset to M/2, but persons of ordinary skill in the art willrecognize that other data compression steps producing other compressionratios may be performed in accordance with the present invention.

As in the embodiment of FIGS. 18A and 18B, the optional interpolationstep in the embodiment of FIG. 19A performed after data expansionincreases the amount of data in the image dataset from M to 4M. Personsof ordinary skill in the art will recognize that the example shown inFIG. 18A is a non-limiting example, and other interpolation processesperformed in accordance with the principles of the present inventionwill result in increasing the amount of data in the dataset by factorsother than 4.

Referring now to FIG. 19B, a variation on the image capture and displaysystem and method of the present invention of FIG. 19A is presented.Because the elements and process steps of the embodiment of FIG. 19B arepresent in the embodiment of FIG. 19A, the same reference numerals usedin FIG. 19A will be employed to identify the corresponding elements andsteps of the embodiment of FIG. 19B.

In the variation of the image capture and display system and method ofthe present invention depicted in FIG. 19B, the compressed RGB datasetfrom imager 232 is stored in storage element 236 without any colortransformation or correction being performed. As may be seen from anexamination of FIG. 18B, the color transformation and/or correction isperformed after retrieval of the image dataset from storage at step 238and data expansion at step 264 and prior to interpolation and display orprinting. Otherwise, the image capture and display system depicted inFIG. 19B may be identical to that depicted in FIG. 19A.

The image capture and display method performed by the embodiment of thepresent invention depicted in FIG. 19B starts with the same step 232 ofthe method of FIG. 19A wherein the image data is captured by the imagerand an image dataset is formed. Next, at step 262, data compression isperformed on the image dataset from imager 232. The compressed imagedata is then stored at step 236.

When it is desired to display or print a stored digital image, thestored dataset representing the image is retrieved from storage at step238. The dataset is then decompressed at step 264. Color correctionand/or transformation is then performed on the retrieved image datasetat step 234. The color corrected image dataset may then be interpolatedat step 240 if desired. Finally, at step 242, the image data is thenprovided to display or printer as required by the user and known topersons of ordinary skill in the art.

While embodiments and applications of this invention have been shown anddescribed, it would be apparent to those skilled in the art that manymore modifications than mentioned above are possible without departingfrom the inventive concepts herein. The invention, therefore, is not tobe restricted except in the spirit of the appended claims.

What is claimed is:
 1. A method for storing a full RGB datasetcomprising: providing an image sensor having a plurality of pixellocations and configured to sense, at the same place in each pixellocation, all three primary colors in order to produce a full RGBdataset as three-color output data; providing a digital storage devicecoupled to said image sensor; sensing three-color digital output datarepresenting said full RGB dataset from said image sensor; and storingsaid three-color output data as digital data in said digital storagedevice without performing any interpolation on said three-color outputdata.
 2. The method of claim 1 wherein providing said digital storagedevice further comprises providing a semiconductor memory device.
 3. Themethod of claim 1 wherein providing said digital storage device furthercomprises providing a magnetic storage device.
 4. The method of claim 1wherein providing said digital storage device further comprisesproviding an optical storage device.
 5. The method of claim 1 furtherincluding performing a lossless compression operation on saidthree-color output data prior to storing said three-color digital outputdata as digital data in said digital storage device.
 6. The method ofclaim 1 further including performing a nearly lossless compressionoperation on said three-color output data prior to storing saidthree-color digital output data as digital data in said digital storagedevice.
 7. A method for processing digital information from an imagesensor comprising: providing an image sensor having a plurality of pixellocations and configured to sense, at the same place in each pixellocation, all three primary colors in order to produce a full RGBdataset as three-color output data; providing a digital storage devicecoupled to said image sensor; sensing three-color output datarepresenting said full RGB dataset from said image sensor; storing saidthree-color output data as digital data in said digital storage devicewithout performing any interpolation on said three-color digital outputdata; and retrieving said three-color output data as digital data fromsaid digital storage device.
 8. The method of claim 7 wherein providingsaid digital storage device further comprises providing a semiconductormemory device.
 9. The method of claim 7 further including: performing alossless compression operation on said three-color output data prior tostoring said three-color output data as digital data in said digitalstorage device; and performing a lossless decompression operation onsaid three-color output data after retrieving said three-color digitaloutput data as digital data from said digital storage device.
 10. Themethod of claim 7 further including: performing a nearly losslesscompression operation on said three-color output data prior to storingsaid three-color output data as digital data in said digital storagedevice; and performing a nearly lossless decompression operation on saidthree-color output data after retrieving said three-color digital outputdata as digital data from said digital storage device.
 11. A method forstoring digital information from a single-chip image sensor comprising:providing a single-chip image sensor producing three-color output datafrom the same place in each of a plurality of pixel locations; providinga digital storage device coupled to said single-chip image sensor;sensing three-color digital output data from said single-chip imagesensor; and storing said three-color output data as digital data in saiddigital storage device without performing any interpolation on saidthree-color output data.
 12. The method of claim 11 wherein providingsaid digital storage device further comprises providing a semiconductormemory device.
 13. The method of claim 11 wherein providing said digitalstorage device comprises providing a magnetic storage device.
 14. Themethod of claim 11 wherein providing said digital storage device furthercomprises providing an optical storage device.
 15. The method of claim 9further including performing a lossless compression operation on saidthree-color output data prior to storing said three-color digital outputdata as digital data in said digital storage device.
 16. The method ofclaim 9 further including performing a nearly lossless compressionoperation on said three-color output data prior to storing saidthree-color digital output data as digital data in said digital storagedevice.
 17. A method for processing digital information from asingle-chip image sensor comprising: providing a single-chip imagesensor producing three-color output data from the same place in each ofa plurality of pixel locations; providing a digital storage devicecoupled to said single-chip image sensor; sensing three-color outputdata from said single-chip image sensor; storing said three-color outputdata as digital data in said digital storage device without performingany interpolation on said three-color digital output data; andretrieving said three-color output data as digital data from saiddigital storage device.
 18. The method of claim 17 wherein providingsaid digital storage device further comprises providing a semiconductormemory device.
 19. The method of claim 17 further including: performinga lossless compression operation on said three-color output data priorto storing said three-color output data as digital data in said digitalstorage device; and performing a lossless decompression operation onsaid three-color output data after retrieving said three-color digitaloutput data as digital data from said digital storage device.
 20. Themethod of claim 17 further including: performing a nearly losslesscompression operation on said three-color output data prior to storingsaid three-color output data as digital data in said digital storagedevice; and performing a nearly lossless decompression operation on saidthree-color output data after retrieving said three-color digital outputdata as digital data from said digital storage device.
 21. A method forprocessing digital information from a triple-junction active pixel arraycomprising: providing a triple-junction active pixel array producing afull RGB image data set; providing a digital storage device coupled tosaid triple-junction active pixel array; sensing the full RGB data setfrom the same place in said triple-junction active pixel array; storingsaid full RGB data set as digital data in said digital storage devicewithout performing any interpolation on said full RGB data set; andretrieving said full RGB data set as digital data from said digitalstorage device.
 22. The method of claim 21 wherein providing saiddigital storage device further comprises providing a semiconductormemory device.
 23. The method of claim 21 further including: performinga lossless compression operation on said full RGB data set prior tostoring said full RGB data set as digital data in said digital storagedevice; and performing a lossless decompression operation on said fullRGB data set after retrieving said full RGB data set as digital datafrom said digital storage device.
 24. The method of claim 21 furtherincluding: performing a nearly lossless compression operation on saidfull RGB data set prior to storing said full RGB data set as digitaldata in said digital storage device; and performing a nearly losslessdecompression operation on said full RGB data set after retrieving saidfull RGB data set as digital data from said digital storage device. 25.A method for processing digital information from a digital camera imagecapture and display system comprising: providing a digital camera havinga triple-junction active pixel array; providing a digital storage devicecoupled to said triple-junction active pixel array; producing a full RGBdata set from the same place in said triple-junction active pixel array;sensing the full RGB data set from said triple-junction active pixelarray into said digital storage device; storing said full RGB data setas digital data in said digital storage device without performing anyinterpolation on said full RGB data set; and retrieving said full RGBdata set as digital data from said digital storage device.
 26. Themethod of claim 25 wherein providing said digital storage device furthercomprises providing a semiconductor memory device.
 27. The method ofclaim 25 further including: performing a lossless compression operationon said full RGB data set prior to storing said full RGB data set asdigital data in said digital storage device; and performing a losslessdecompression operation on said full RGB data set after retrieving saidfull RGB data set as digital data from said digital storage device. 28.The method of claim 25 further including: performing a nearly losslesscompression operation on said full RGB data set prior to storing saidfull RGB data set as digital data in said digital storage device; andperforming a nearly lossless decompression operation on said full RGBdata set after retrieving said full RGB data set as digital data fromsaid digital storage device.